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Abstract 


International surveys, such as TIMSS and PISA, frequently put Indonesia in the low ranks. It is an indica- 
tion that the higher-order thinking skills (HOTS) of students in Indonesia are still low. This research aims 
to analyze students’ difficulties in solving problems that measure HOTS. This is a case study research with 
a qualitative approach. Participants studied were 93 high school students in grade XI. Data were col- 
lected using test instruments that measure HOTS, which was developed based on the standard contents 
of high school mathematics. The difficulties were analyzed descriptively by observing students’ errors in 
answering HOTS test items. Students’ errors were classified based on Newmans Error Procedure (NEP). 
The result shows that around 8.33% of the students had difficulties in comprehension, 15.59% in trans- 
formation, 32.53% in process Skills, and 1.34 % in encoding. 

Keywords: HOTS problem in mathematics, students’ difficulties, case study, Newmans error procedure. 


Introduction 


Education nowadays cannot be viewed merely as transferring knowledge from teachers 
to students. In modern concept, education becomes one of the means for preparing students to 
encounter every challenge of the modern era. In response to the global challenges, there have 
been adjustments in the curriculum of education in Indonesia by taking into account the current 
global challenges. Implementation of the 2013 curriculum in Indonesian education system is 
now beginning to update the direction of learnings, one of them focusing on the development 
of high-order thinking skills (HOTS) (Ministry of Education and Culture, 2016). Not only 
in learning activities, HOTS test items are now also tested in the National Exam (UN), even 
though in particular, the achievement of the HOTS test items is far from satisfactory (Ret- 
nawati, Kartowagiran, Arlinwibowo, & Sulistyaningsih, 2017). This has shown the seriousness 
of HOTS that it is not only a discourse in the concept of education in Indonesia, but it can be 
viewed as one of the main goals of the educational process. 

Early adaptation of HOTS development in Indonesian education system has demon- 
strated that HOTS is urgent and relevant to current global needs and challenges (Apino & 
Retnawati, 2017; Jailani, Sugiman, & Apino, 2017; Yen & Halili, 2015). Operationally it will 
be easier to define HOTS when referring to cognitive taxonomies. In Indonesia, the cognitive 
taxonomies which already being used in the formulation of learning objectives are the Revised 
of Bloom’s taxonomy. In the taxonomy, HOTS includes the cognitive dimension of analyzing, 
evaluating, and creating (Brookhart, 2010; Liu, 2010). In this regard, the use of the revised of 
Bloom’s taxonomy is more relevant to current educational demands, therefore HOTS can be 
defined as cognitive skills in the analyzing, evaluating and creating levels. 
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HOTS as a cognitive skill is highly needed by students in every education levels and 
courses, one of which is mathematics course. HOTS 1s a skill that must exist in every lesson 
(Sulaiman, Muniyan, Madhvan, Hasan, Rahim, 2017). The implementation of HOTS in math- 
ematics learning is intended to change the perception that mathematics is a difficult subject, 
and to attract learners’ attention to study mathematics (Abdullah, Mokhtar, Halim, Ali, Tahir, & 
Kohar, 2017). Moreover, HOTS plays a role in the training of logical and critical thinking, as 
well as reasoning skills, that are fundamental to everyday life. These skills set are also parts of 
academic achievement (Marshall & Horton, 2011). It shows that HOTS is an important aspect 
of the learning process and students’ HOTS achievement needs to be evaluated to determine the 
extent of students’ HOTS in the learning process. 

Various International studies, such as TIMSS and PISA can be used as a reference to de- 
termine the HOTS achievement of the students (Budiman & Jailani, 2014; Jaelani & Retnawati, 
2016; Apino & Retnawati 2017). In the TIMSS 2015, for mathematics skills, Indonesia was 
in the lower ranks and students’ skills were still in the ‘knowing’ and ‘applying’ domain or in 
lower order thinking skills. In the ‘reasoning’ domain, students skill still was very low (Mullis, 
Martin, Foy, & Hooper, 2016). Similarly, in the PISA 2015, Indonesia was in the 115™ position 
out of 124 participating countries. Indonesian students could only perform in the 1“ to 4" levels, 
while they still had difficulties in the 5" to 6" levels (OECD, 2016). The results of these studies 
have provided information that students’ HOTS in Indonesia is still low. 

The low students’ HOTS skills have proved that students faced difficulties in solving 
mathematics test items that measure HOTS. Analysis of the students’ difficulties in solving 
HOTS test items is essentially the same as the analysis of students’ difficulties in solving math- 
ematics test items in general. In this case, students’ difficulties can be identified by analyzing 
students’ errors in solving mathematics test 1tems (Wiaya, van den Heuvel-Panhuizen, Door- 
man, & Robitzsch, 2014). One of the procedures that can be used to identify students’ errors in 
solving HOTS test items is Newman’s Error Analysis (NEA) (Wijaya van den Heuvel-Panhui- 
zen, Doorman, & Robitzsch, 2014; Abdullah, Abidin, & Ali, 2015). Newman (1977) stated that 
students’ errors in solving mathematics test items involve reading errors, comprehension errors, 
transformation errors, process skills errors, and encoding errors. The data of students’ errors can 
be used to identify students’ difficulties in solving test items that measure HOTS in mathematics 
learning (Abdullah et al., 2015). 

Many researches have been carried out to investigate students’ difficulties in solving 
mathematics test items. Jupri and Drijvers (2016) had investigated students’ difficulties in solv- 
ing algebra word (story) problems and the result showed that the student encountered difficul- 
ties in formulating the mathematical model, indicated by errors in mathematical equations, 
schematics or diagrams. Phonapichat, Wongwanich, and Sujiva (2014) had identified students’ 
difficulty in solving mathematics problem, namely difficulty in understanding problems’ key- 
words, finding assumption and key information in the problem, wild guessing when student 
failed to understand the problem, impatience and reluctance to read the problem thoroughly, 
and students dislike long word problems. Alhassora, Abu, and Abdullah (2017) found that the 
difficulty in solving HOTS test items was because the students are not accustomed to reading 
long word problems and due to the low reading skills of the students. These difficulties are the 
cause of many errors made by the students in answering HOTS test items. 

Prakitipong and Nakamura (2006) studied students’ error in solving mathematics test 
items in Thailand. The result showed the types of students’ errors, namely comprehension er- 
rors in solving structured questions and transformation errors in solving multiple-choice ques- 
tions. Abdullah et al. (2015) studied students’ error in solving HOTS and found that students’ 
errors consist of encoding, process skills, transformation, and comprehension. Students also 
had difficulties in relating information and implementation of strategies to solve HOTS test 
items. Similar findings concluded by Santoso, Farid, and Ulum (2017) showed that students’ 
error type in solving word problems of linear programs are transformation errors and process 
skill errors. 
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Research Focus 


Based on the background and theoretical studies that have been previously described, it 
is important to identify the difficulties in solving mathematics problems to measure students’ 
HOTS. Thus, the purpose of this research is to analyze the type of difficulty experienced by stu- 
dents in solving mathematics questions that measure HOTS. The findings of this research can 
be used as an input for teachers to design the best practices in teaching mathematics to improve 
students’ HOTS. 

This research is different from the research on the analysis of students’ difficulties in 
completing the PISA question that have been done by van den Heuvel-Panhuizen, Doorman, & 
Robitzsch (2014). Their research emphasized the context-based PISA mathematics tasks per- 
formed in junior high school. This study emphasizes the difficulties of students in solving the 
HOTS problems in senior high school, although both of these researches are using NEP. 


Methodology of Research 
Type of Research 


This research is a case study aimed to analyze students’ difficulties in solving mathemat- 
ics problems that can measure HOTS, using quantitative and qualitative approach. It was con- 
ducted in May — October 2017. In this research, the researcher identifies the types of difficulties 
experienced by students when solving mathematics problems that measure HOTS. Classifica- 
tion of student difficulties refers to Newman's Error Procedure (NEP) which consists of reading 
errors, comprehension errors, transformation errors, process skills errors, and encoding errors. 


Participants of Research 


The research participants were 93 students (about 77.22% students) of grade XI of Sci- 
ence Program at SMAN | Seyegan (Senior High School), Special Region of Yogyakarta, In- 
donesia. These participants consisted of 30 male and 63 female students with heterogeneous 
academic skills (low, medium, and high). These subjects were assumed that have learned the 
tested learning material in the research instruments. 


Instrument and Procedures 


Data were collected using multiple-choice test, in which the research subjects were 
asked to write down the steps of their answers. The test consists of 3 items about probability 
and statistics, 2 items about limit in function, 4 items about differential, and 5 items about the 
applications of differential, and the total is 14 items. These items are structured representatively 
to measure the content of the material in mathematics for one semester in class X of senior high 
school. It means the test considered logical validity. The reliability of test’s score estimated us- 
ing alpha-Cronbach coefficient, that is 0.833 (high category). 

From the 14 items, the researchers have chosen 4 items. The 4 items measure HOTS that 
are problems 1-4. All of the items are about the application of differential. These problems are 
presented in the result together with the analysis. 

Using students’ responses to the tests, the students’ difficulties were identified. The iden- 
tification of the students’ difficulty types was observed from the errors made by the students in 
answering the test items. The types of errors were categorized based on Newman’s Error Pro- 
cedure (NEP), they are reading, comprehension, transformation, process skills, and encoding 
errors. These errors were identified independently, so in this case, students probably can make 
any mistakes more than one times. 
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To maintain the credibility of the obtained data, the test was done with the exam exercise 
format. In this format, students finished the test wholeheartedly, so that the answers provided 
the reflection of the students' true abilities. The researchers also informed that the result of stu- 
dents’ difficulties identification would only be used to give feedback of improvement, will not 
determine the students’ positions. To maintain objectivity during checking students’ answers, 
researchers ignored the students’ identities. 


Data Analysis 


Data were analyzed quantitatively and qualitatively. The quantitative analysis was car- 
ried out to determine the percentage of each difficulty encountered by the students in solving 
HOTS test items. The qualitative analysis was carried out to analyze the students’ error pattern 
based on NEP. The analysis of data included the steps of scoring students’ answers, classifica- 
tion of error types based on NEP, calculating the percentage of each of the difficulty type, and 
performing a qualitative analysis to describe each of the students’ difficulty. 

Using students’ responses to the items, researchers check and count the errors done by 
students. In every item, a student could do more than one error. For example, in Problem 1, a 
student has done an error in comprehension and transformation. Then, in every kind of error 
in NEP, the researchers calculating the percentage, how many students that have done in every 
kind of error divide with the total of research participants. 


Results of Research 

Data of students’ difficulties in solving HOTS test items were identified using Newman’s 
Error Procedure (NEP). Also, identified in this research was the number of students who gave 
the correct answers and those who did not answer. The summary of students’ errors analysis in 


solving HOTS test items is presented in Table 1. 


Table 1. Summary of the results of students’ error analysis. 


Problem 1 Problem 2 Problem 3 Problem 4 Sue 
n % n % n % n % % 
Correct Answer 2 Zi 38 40.86 23 24.73 0 0 16.94 
Not Answer 56 60.22 27 29.03 30 32.26 45 48.39 42.47 
Reading 0 0 0 0 0 0 0 0 0 
Comprehension 10 15° 5 5.38 = 13 13.98 3 3.23 8.33 
Transformation 11 11.83 11 11.83 26 27.96 10 1075 15,59 
Process Skills 34 36.56 23 24.73 24 25.81 40 43.01 32.53 
Encode 0 0 1 1.08 2 215 2 210 1.34 


Table 1 informs that the students were still having difficulties in working on the problems. 
It is indicated by the low percentage of students with correct answers (an average percentage of 
16.94%), and even for Problem 4, there are none who had answered it correctly. In addition, the 
percentage of students who did not answer questions was also quite high (an average percent- 
age of 42.47%). This is a strong indication that the students had difficulties in solving the given 
test problems. A graph of students’ errors in solving HOTS test items is presented in Figure 1. 
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Figure 1. Error distribution of the students in solving HOTS test items. 


Based on Figure 1, Problem 2 is the most correctly answered problem whereas, for Prob- 
lem 4, none of the students could answer it correctly. The number of students who did not an- 
swer was quite high, in which Problem 1 was the most unanswered test item. Comprehension 
errors were experienced mostly in solving Problem 3, and so were the transformation errors. 
Process skill errors were the most common error experienced by the students compared with 
other error types and mostly found in Problem 4. In addition, encoding errors were the least 
experienced errors by the students and commonly encountered in working on problem 3 and 
4. The next section will describe the error types experienced by the students in solving HOTS 
test items. 


Comprehension Errors 


Comprehension errors are marked by students’ errors in the identification of “what is 
asked” and “what is given” from the test items. This error also can be marked by students’ in- 
ability to write down important and relevant information of the test item. Figure 2 shows an 
example of students’ comprehension errors. As seen in Figure 2, students comprehend the test 
sentence “the product of the squares of the two numbers” as y = x’ : y’ (the one being circled), 
which should have been written as x’ «y= 0. Another example of students’ comprehension error 
is presented in Figure 3. Student’s answer in Figure 3 indicates that the student was having dif- 
ficulty in comprehension of the test item. The written answer did not represent what was asked 
and what was given in the test item, it did not even lead to the correct answer. This is a strong 
indication of the student’s difficulty in comprehension of the test item. 
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Question: Problem 3 Student response: 
The sum of two numbers 1s 8. In the 

event that the product of the squares of the 

two numbers reaches the maximum value, 

the difference between the largest and the 

smallest numbers is ... (Answer = 0) 





Figure 2. Example of comprehension error. 


Question: Problem 4 Student response: 
A door has a similar geometry 
to the figure below. 


2x 
If the circumference of the door 
is p, the value to make the area of the 
door at maximum is ... . (Answer = 


ore, 





Figure 3. Example of comprehension error. 
Transformation Errors 


Transformation errors are marked by students’ error in developing a mathematical model 
from the information given, error in selecting the formula to be used, and error in the comple- 
tion of the test item. An example of students’ transformation error is presented in the Figure 4. 
Figure 4 shows a fraction of student’s answer in solving Problem 1. The figure shows that the 


student used a formula of = * circumference of the big square to find the solution. The solu- 
tion plan was wrong; the student should directly calculate the circumference of the small square 
because the diagonal of the small square can be determined. Because of the error in choosing 
the solution plan, the student was unable to find the final solution for the test item. This indi- 
cates that the student made an error in planning the solution of the test 1tem, and thus indicates 
that the student was still encountered difficulty in the transformation of the test item. 
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Question: Problem 1 Student response: 


Consider the following figure. 
O 


a’ 

A square ABCD has diagonal BD = 
124/2 cm and £ CBD = x*. If the square is 
scaled into half the length and half the width, 
twice, as shown in the figure, then the maxi- 
mum circumference of the smallest square is 
... . (Answer = 12 cm) 





Figure 4, Example of transformation error. 


One more example of students’ transformation error is presented Figure 5. Figure 5 
shows that student made a mistake in determining the formula of a rectangle’s circumference. 
In Figure 5 (the circled portion) it 1s seen that to determine the circumference of the rectangle 


the student used a formula of P X ! (length times width), which in fact this formula should only 
be used to calculate the area of a rectangle. Due to this error, the student failed to find the solu- 
tion to the given problem. This strongly indicates that there were students who still encountered 
difficulties in transforming the test item. 


Question: See Problem 4 
Student response: 





Figure 5. Example of transformation error. 
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Process Skill Errors 


Process skills errors are marked by students’ error in implementing a formula, error in 
math calculation, error in algebraic manipulation, and error in implementing mathematical pro- 
cess in solving a problem. This research found that process skill error is the most common error 
made by the students and it happened in the entire test 1tems. Figure 6 describes an example 
of student’s process skill errors. The First error is the error in algebraic operation, in which 


the function 3x — 990 + =" (step 1) should be multiplied by the x variable, which would produce 


3x2 —900x +120, but the student’s process (as can be seen in step 2) resulted in ——E 
It is obviously erroneous and indicates that the student was having difficulty in the algebraic 
multiplication operation. Furthermore, looking closely at step 3 in Figure 6, the factorization 
performed by the student still resulted an error. It can be concluded that students’ process skill 
in algebraic operation is still very low. These findings indicate that student still encountered 


difficulties in mathematical process skill, especially in algebraic operation. 


Question: Problem 2 Student response: 
Construction project of a school 
building can be completed in x days, with 


project cost per day as (3x — 9004 22) in 
hundred thousand rupiahs. In order to 
make the cost at minimum, the project 
should be completed in ... days. (Answer 
= 150) 


Figure 6. Example of process skill error. 


Here in after, another example of process skill error is presented in Figure 7. 
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Question: See Problem 3 
Student response: 


= 





Figure 7. Example of process skill errors. 


Students’ error shown in Figure 7 happened in step 3 and 4, in which the student made an 
error in performing the algebraic multiplication operation. Furthermore, it can also be seen that 
the student made an error in the manipulation of algebraic form, as shown in step 6. The student 
also managed to make another error in the derivation of the algebraic manipulation (step 7). 
This indicates that students were still having difficulties in process skill related to an algebraic 
manipulation operation. 


Encoding Errors 
Encoding errors are marked by students’ error in defining conclusion. Encoding error is 


the least error type made, in which only 1.34% of the students were experiencing this error type. 
An example of student’s encoding errors in solving Problem 3 is presented in Figure 8. 





Figure 8. Example of encoding error. 


Figure 8 shows a process done by the student in solving Problem 3. In step 1 the student 
developed a mathematical model from the given information, and in step 2 the student substi- 
tuted one of the mathematical models into the other mathematical model to form a function. 
In step 3 the student determined the derivative of the function and determined the variable’s 
value that satisfies the function when the derivative equals zero. In this case, the student’s error 
happened when determining the difference between the largest and smallest values, which led 
to the multiplication of the numbers’ square at its maximum. From the obtained variable values 
(a = 0, then b = 8; a= 4, then b = 4; and a = 8, then b = 0), the student should have tested first 
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the multiplication that will lead to a maximum value. In this case, the student only looked at the 
value that was maximum, which was 8, and opted for option C. 


Discussion 


Results of data analysis show that the errors done by the students in solving HOTS test 
items are comprehension error, transformation error, process skill error and encoding error. The 
research findings are in line with the results obtained by Abdullah et al. (2015) and Santoso 
et al. (2017). These two researches both stated that errors experienced by students in solving 
mathematics problems are comprehension error, transformation error, process skill error, and 
encoding error. These errors serve as an indicator that students are still having difficulties in 
solving mathematics problems that measure HOTS. From the errors type stated by Newman 
(1977), the data analysis result shows that there is one type of errors which was not encountered 
by the student in solving the test problems, namely reading error. This finding is also in line 
with the result of Abdullah et al. (2015) and Santoso et al. (2017). Their researches both show 
that none of the test subjects had difficulty in reading the test problems. It occurs because the 
subjects were grade XI junior high school students, who already have good reading skills. In 
addition, the type of test problems came with figures or illustrations, so that the information 
in the test problems could be understood almost instantly just by looking at those figures or 11- 
lustrations. 

The crucial difficulties experienced by the students in solving test problems were process 
skills and transformation difficulties. Difficulty in mathematical process skills in this research 
mostly happened when students performed an algebraic operation and manipulated algebraic 
forms. This is due to their low understanding of the concept of algebraic operations and the 
manipulation of algebraic forms, and lack of ability in mathematical connections. The lack of 
students’ conceptual understanding becomes one of the constraints in solving HOTS test items 
(Alhassora et al., 2017). Likewise, in performing transformation, there were still a few students’ 
made errors in determining the formula to answer test problems. This is a concrete proof that 
not all of the students have a good understanding of the concept. In addition, students’ error in 
determining the mathematical model of a problem only proved that their transformation skills 
were still low. Rachmawati (2016), Jupri and Drijvers (2016) stated that one of the constraints 
the students experienced in solving mathematics is the difficulty in developing the mathemati- 
cal model. This should be used as a reflection for math teachers to improve the forthcoming 
math teaching. In this case, there are two pivotal points that should be the focus of math teach- 
ers, which are strengthening the conceptual understanding for students, especially in algebra, 
and training the students in developing the mathematical model. 

Comprehension of the problem is another difficulty experienced by students in answer- 
ing HOTS test items. An indication of this difficulty can be seen from students’ error in writing 
down what is given and what is asked on the test items. In addition, students still often made 
errors in selecting and writing down important information and relevant facts from the prob- 
lem. It can be identified from the students’ answer sheets that one of the causes of difficulty in 
comprehension is that students were not accustomed to tests items that led them to an in-depth 
analysis. For the comprehension of the problem’s context, students preferred short questions 
and explicitly written problems. Therefore, when presented test items that required in-depth 
analysis, students tended to experience difficulties. This finding 1s supported by the results of 
Phonapichat et al. (2014) research, which found that students are often impatient and do not like 
to read the passage of the test items. 

The high percentage of students who were not answering was also an evidence of stu- 
dents’ low interest in HOTS test items. This also indicates that the students were not challenged 
by HOTS test problems. It happened because they were not accustomed to HOTS model test 
items. Because students were not accustomed to HOTS model test items, they were confused 
when working on the answers. They were not even capable to determine what 1s given and what 
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is asked on the test items. Other than that, the low perseverance of the students really affects 
how they answered. Students with low perseverance tended to work on less challenging prob- 
lems and the ones that had a clear workaround. If a problem is deemed as difficult by the stu- 
dents, there is a high chance that it will not be answered. Students’ literacy also plays a role on 
whether a problem will or will not be answered. Phonapichat et al. (2014) stated that students 
dislike long word problems. Therefore, when presented with test items that encourage them to 
use literacy skills, students tend to be reluctant in solving them. 

Many efforts can be implemented to overcome the difficulties experienced by students. 
Teachers can strengthen the understanding of mathematical concepts during math lessons (Ret- 
nawati, Kartowagiran, Arlinwibowo, Sulistyaningsih, 2017), make improvements in teaching 
and learning process (Retnawati, Munadi, Arlinwibowo, Wulandari, Sulistyaningsih, 2017), 
utilize various media and technology in teaching and learning (Retnawati, 2015), also utilize 
the right learning trajectory during the learning process (Retnawati, 2017; Retnawati, Arlinwi- 
bowo, Wulandari, Pradani, 2018) and use HOTS problem in assessment (Retnawati, Hadi, Nu- 
graha, 2016). Improving teacher qualification in mathematics teaching and learning (Retnawati, 
Dyjidu, Apino, Kartianom, Anazifa, 2018) 1s also an attempt to make students’ achievements 
better. 


Conclusions 


The most common difficulty for students in solving test items that measure HOTS is 
math process skills. This difficulty is indicated by errors in implementing the formula, errors 
in mathematics calculation, and errors in algebraic operation and manipulation. Next in line is 
the difficulty in the transformation of the problem, which was indicated by students’ error in 
developing a mathematical model, errors in determining the formula to solve the problem, and 
difficulty in planning to solve the problem. Other difficulties experienced by the students are 
difficulties in comprehension of the test item, which was indicated by students’ error in deter- 
mining what is given and what is asked on the test items. Encoding is also one of the difficulties 
experienced by students. This can be indicated by errors in interpretation of the result obtained. 
The difficulties mentioned earlier which are experienced by the students are caused by many 
factors. First, students are not familiar with the tests that measure HOTS. Second, the low per- 
severance of the students in solving HOTS test items. Third, there is a tendency for students to 
dislike long word problems. Teachers’ role is very important to improve students’ capability in 
solving HOTS test items. In this case, teachers need to train their students so that they will be 
familiar in solving HOTS test items. 

Other researches related to student difficulties in solving the HOTS problem in math- 
ematics can be done. The students' habits to solve the HOTS problem need to be established 
along with the development of appropriate mathematical teaching and learning strategies or 
methods. This teaching and learning process needs to be supported by the development of 
HOTS media or teaching aids. Supporting students’ abilities to resolve mathematics problems, 
such as conceptual understanding, mathematical connections, and mathematical representa- 
tions abilities also need to be identified. 
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